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ABSTRACT 

This paper describes a series of experiments conducted 
within the context of a course on organizational theory that is taught at the 
Department of Management Sciences at the University of Twente (Netherlands) . 
In 1997, a group-based learning approach was adopted, but after the first 
year it was apparent that acquisition and application of theory by student 
groups was inadequate. In an attempt to remedy this problem, a World Wide 
Web-based collaborative work environment was introduced in 1998 with the 
intention of encouraging students to read relevant theoretical material and 
to reflect more on what they had read. In addition to hosting a theory 
repository, the collaborative work environment was designed to control the 
flow of work and to enforce rules for groups 1 access to the output of other 
groups, based on their own performance. Further changes were made, and a 
third edition of the course was run and evaluated in 1999. A description of 
the educational setting and the Web-based theory repository is presented. The 
evaluation results over the period 1997-99 are presented and discussed. The 
extent to which the discipline of reading improved was evaluated, as were the 
effects on insight into theory. It turns out that the technical realization 
works well. Uptake of the instructional tasks for reflection, however, only 
takes place if these tasks are perceived as being helpful. (Author) 
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to the educational resources Abstract: This paper describes a series of experiments conducted within the context of a 

information center (ERIC) course on organisational theory which is taught at the Department of Management 

Sciences at the University of Twente. In 1997 a group-based learning approach was 
adopted but after the first year it was apparent that acquisition and application of theory 
by student groups was inadequate. In an attempt to remedy this problem a Web-based 
collaborative work environment was introduced in 1998 with the intention of 
encouraging students to read relevant theoretical material and also to reflect more on 
what they had read. In addition to hosting a ‘theory repository’, the collaborative work 
environment was designed to control the flow of work and to enforce rules for groups’ 
access to the output of other groups, based on their own performance. Further changes 
were made and a third edition of the course was run and evaluated in 1999. A description 
of the educational setting and the Web-based "Theory repository" is presented. The 
evaluation results over the period 1997-1999 are presented and discussed. The extent to 
which the discipline of reading improved was evaluated, as were the effects on insight 
into theory. It turns out that the technical realisation works well. Uptake of the 
instructional tasks for reflection, however, only takes place if these tasks are perceived as 
being helpful. 



* Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



I/) 

o 

o 




Educational Setting 

The course “People, Technology and Organisation-2” is given to 150-200 undergraduate students. This 200 
hours course focuses on organisation theory and its relevance for designing business organisations. Since 
1997 a problem-based learning approach was adopted (Barrows & Tamblyn 1980) with the intention to 
activate and motivate the students with authentic study-tasks. The students work in project groups each 
consisting of six or seven students. Over a period of 10 weeks the groups study theory and work on 
exercises that come with pre-defined case studies. The theoretical component consists of a textbook and 3 
sets of 8 theory articles. The case studies refer to a number of organisational issues in car manufacturing. 
Parallel to the group work there are a number of lectures with small groups (40 students). In the lectures the 
student groups present their findings, after which discussion takes place moderated by an instructor. The 
final mark for the course is a combination of the individual mark for the textbook exam and the group mark 
for case reports. The 1997 course set-up was evaluated by (Smit & Riemsdijk 1998), exposing flaws in the 
theory part of the group work. In many groups, the transfer of what students have read individually to the 
group members was poor. This resulted in little use of relevant theories in the case-study reports by the 
groups. The question investigated in our experiments is thus: Can Web-support activate the use of theory in 
an efficient way ? The planned relation between the learning goals and the workforms is given in (Table 1). 



Learning goal 


Theory & 
Web-site 


Lectures & 
discussion 


Group work & 
discussion 


Knowledge acquisition 


++ 






Improving insight 


+ 


++ 


+ 


Application in problem solving 




+ 


++ 



Table 1: Relating course components and learning goals (++ primary goal, + secondary goal). 
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Design of a Web-Based Theory-Repository 



To stimulate the reading of theory articles and reflection on theoretical issues, a Web-site was set up in 
1998 with the aim of forming a "Theory-repository". The division of reading tasks was left to the groups. 
For each of the 24 articles on organisational theories every group had to submit a contribution which 
targeted the core of the article. This took the form of two questions about the article together with model- 
answers. The contributions of a group form a group resource. The set of contributions on a particular article 
together form a resource for all groups. This resource is only made accessible to another group after that 
group has submitted a serious contribution on that same article themselves. This basic quality assessment 
(group contribution is "not ok" or "ok") was performed by teaching assistants. The students also ranked the 
submissions of other groups, indicating per article which 5 contributions they judged to be of highest 
quality. This top-5 exercise was intended to further enhance reflection on theory. The 1998 evaluation 
results (Veen, Riemsdijk, Slabbekoom & Kamp 1999) show that the discipline of reading theory articles 
had been enhanced. However the students felt that the formulation of questions and model answers did not 
help them very much in their group work. The students reported that, after finishing this assignment, they 
still had to work out summaries for their group. The top-5 assignment was felt to be "a waste of time" as 
reading sometimes more than 20 contributions on the same article took too much time and gave the 
students little added value. This top-5 assignment was partly ignored by the students, and finally skipped by 
the instructors on the last set of 8 theory articles. 

Based on the experiences of 1998, a revised setup for the Web-site was introduced in 1999. Instead of 
questions and model answers, groups were now asked to contribute a summary per article. The summaries 
per group thus form a collection that the groups really need in their sharing of expertise. Ranking was now 
included in the grading by teaching assistants (group contribution is "not ok", "ok" or "excellent"). Instead 
of the obligatory top-5 assignment, the students were now offered the option to read a small number of 
excellent summaries by other groups, again only after a serious contribution on that same article by the 
group themselves. The goal of this re-designed cross-group exchange of expertise was to stimulate more 
efficient reflection on theory. (Figure 1) shows the relations between the main concepts in a UML class 
diagram (Larman 1998). 



Course 


* 


lectured by > 


1..* 


Instructor 










Shows only contributions withl^ 
Assessment excellent=true 



Access only for groups with 
Assessment ok=true 



ok : boolean 
excellent : booleaifi 
comment : string 



Cardinality legenda: ^ 
* zero or more 

0..1 zero or one 
1 exactly one 

1..* one or more 



Figure 1: UML class diagram of the 1999 version of the "Theory Repository". Reading directions of 
relations is indicated by symbols ( A ,<,>). 
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Ergonomics design was improved in 1999 by analysing the sequences of mouse-clicks required for basic 
actions like "reading group contributions ", "reading contributions of other groups", and "assessing 
contributions" (Veen, Riemsdijk, Jones & Collis 2000). When students log in to the course site a 
personalised user interface is given (Figure 2), offering only options that are accessible to the user. 
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Figure 2: Example of a personalised screen offering conditional access to excellent work of other students. 



Technical Realisation 

Microsoft Internet Information Server was used as the Web-server. Group contributions were stored in an 
MS-Access database, accessible to the server through ODBC (Open DataBase Connector). Active Server 
Page (ASP) scripting was used to implement writing to and querying the database. All relevant ASP-code 
uses ODBC-calls, making the application database independent. Because the operating system (Windows 
NT) does not provide a hierarchical group mechanism, a dedicated user administration add-on using 
ASPUser was set up. This solution exposes the full Windows NT user administration and the file ACL’s 
(Access Control Lists) to ASP and allows user management via a Web-browser. This permits the 
formation of groups at the course and project group level, as well as the implementation of roles and 
accompanying privileges. 



Evaluation approach 

This experiment can be considered as a time-series based case study in a natural setting. The 1997 course 
had no Web-support and that year's evaluation data can be used as a baseline. In 1998 telematics support 
was introduced. Finally in 1999 the telematics support evolved as described above. The main goal of the 
implementation and therefore the main evaluation questions involve the value of the Web-site in improving 
reading discipline, Question 1 (Table 2). The improvement of insight was thought to be reinforced by 
realising a number of instances for reflection through cross-group reading of other groups’ work (Question 
2). The planned for reflection was hopefully perceived as being helpful by the students (Question 3). To see 
if minimal conditions for proper use are met, the ease of use (Question 4) was checked. Also we looked for 
indicators of improved student results (Question 5). Evaluation data are gathered in different ways. The 
triangulation strategy (Stake 1995) is given in (Table 2), indicating for each of the questions the primary 
source of information, as well as additional sources for confirmation or contradiction. 



Page 559 



BEST COPY AVAILABLE 



Question 


Primary source 


Additional source(s) 


Ql Does the Web-site help improve the 
discipline of reading theory? 


Theory repository 


Student interview 


Q2 Are students reading other groups work? 


Event log 


Student questionnaire 
Student interview 


Q3 Is reading other groups work perceived as 
helpful by students? 


Student questionnaire 


Student interview & 
student questionnaire 


Q4 Was the Web-site easy to use? 


Student questionnaire 


Key-stroke analysis 


Q5 Does the Web-site improve student results? 


Instructor interview 


Course grades & event log 



Table 2: Evaluation questions and evaluation methods. 



Results 

We assume the number of submitted contributions to be an indicator for the discipline of reading (Ql). 
(Table 3) indicates that for most of the 24 articles work is submitted in 1998. The raise in 1999 (23.2 > 
26.7) is mainly caused by groups that submitted a new summary in 1999 after having received a ’not ok' 
assessment for their first submission. The teaching assistants approved 67.3 % of the contributions in 1998, 
versus 68.4 % of the contributions in 1999. The percentage of "Excellent Lists" for which groups received 
access is clearly higher in 1999 (76.1%) compared to 1998 (65.0%). This is caused by the additional 
submissions in 1999. In the interview, students indicated that they read the excellent contributions, to get 
information about the content of the different articles, to see how other groups are doing, and to get an 
indication of what an excellent summary should look like. Students prefer to read just the excellent 
summaries instead of all the summaries, because this would take too much time. Also, students indicated 
that they would like to have access to a summary made by the teacher. 



Theory repository statistics 


1998 


1999 


Number of submitted 
contributions per group 


23.2 
s.d =3.7 


26.7 
s.d. =3. 7 


Percentage of approved 
contributions 


67.3 

s.d =11.6 


68.4 
s.d =8.0 


Approved contributions per 
group in % of 24 articles 


65.0 

s.d =15.5 


76.1 

s.d.= 12.2 



Table 3: Group means and standard deviations (s.d.) for the submitted contributions through the Web-site. 

The use of all collected contributions by other students was analysed (Q2). In 1998 the event mechanism 
showed a total of 309 reading events, an average of 1 1 events per group. The number of events decreased 
with time when the top-5 assignment was cancelled. In 1999 this number has risen to 1465 events, an 
average of 64 events per group. Whereas in 1998 per article only one event per group was logged at the 
most, in 1999 an average of 4 individual group members were logged using the option to read excellent 
summaries. The score on item 1 in (Table 4) confirms our analysis. 



Items on student questionnaire 
(l=disagree, 5=agree) 


1998 

n=83 


1999 
n=l 10 


Significance 
1999/ 1998 


1 I always read the contributions on other 
articles. 


1.97 

s.d ,= 1.10 


3.37 

s.d = 1.42 


p< 0.001 


2 If I wanted to submit data through the Web- 
site, it was clear how to do this. 


4.15 

s.d = 1.00 


4.45 

s.d. =0.91 


p < 0.05 


3 On the Web-site, I had a good overview of 
what was finished, and what still to be done. 


3.52 

s.d =1.21 


3.66 

s.d. =1.16 


not significant 


4 It was fun to work with the Web-site. 


2.35 

s.d.= 1.12 


3.03 

s.d =1.12 


p< 0.001 



Table 4: Comparing appreciation of the Web-site of 1998 and 1999 (means and standard deviations). 
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To have an indication about the helpfulness of the Web-site support (Q3), the students were asked to give 
their opinion about the different workforms in relation to the relevant learning goals, see (Figure 3). For 
this 5-point scale questions in a questionnaire were used. In comparison with 1998, the 1999 Web-site 
shows significant higher appreciation scores (p<0,001) for its contribution to knowledge acquisition and 
insight improvement, and a slightly higher score for its contribution to the application of knowledge 
(p<0.05). The small group lectures are clearly highly appreciated. Also the group discussions are perceived 
as important for learning purposes. In the interview with a panel of students from different groups, the 
students indicate that the fact that not everybody reads every article, creates an interdependency that has 
positive effects on discussion and collaboration. The production of summaries for each other was thought 
to be a highly relevant task, as these summaries are a good introduction for those group members not 
having read the article. Access to other group's work also allowed students to compare their work with that 
of others. The score for item 4 (Table 4) supports our conclusion that the opinion of the students with 
respect to the Web-site has shifted from negative to neutral. 



Importance of vwkforms for 
knowledge acquisition (1=Jow, 5=high) 




Importance of workforms for 
enhancing insight (1=tow, 5=high) 







k- ” 


“♦-Theory 

examination 

-©-Group 

discussions 

-A- Class 
sessions 

Website 


a » * m 




XT' 



1997 1998 1999 



Figure 3: Student appreciation of the importance of workforms for knowledge acquisition and enhancing 
insight Scores on a 5-point scale: l=low, 5=high. 1997: n=l 10; 1998: n=83; 1999: n=l 10. 



The Web-site was easy to use (Q4) according to the students, see item 2 (Table 4). Keystroke analysis 
(Veen, Riemsdijk, Jones, Collis 2000) shows that some tasks need less than halve the number of mouse- 
clicks in 1999 compared to 1998. The status information, item 3 (Table 4), will be further enhanced in the 
next edition of the course. It is difficult to quantify the effects on the final products of the groups (Q5). 
Instructors indicate that it is very hard to relate outcomes to specific learning events. The course design is 
an integration of different activities, that they think is now a strong combination. An analysis was carried 
out to check for correlation between two Web-site parameters and the theory examination and case-study 
grades, see (Table 5). The calculations were performed at the group level. 



Pearson correlation values 


Theory grade 


Case-study grade 


1998 (n=28) 


1 999 (n=23) 


1998 (n=27) 


1999 (n=23) 


Cross-group reading 
(number of reading events) 


-0.23 

not significant 


0.01 

not significant 


0.11 

not significant 


-0.04 

not significant 


Quality of contributions 
(percentage approved contributions) 


-0.41 
p < 0.05 


0.55 

p < 0.01 


-0.09 

not significant 


0.07 

not significant 



Table 5: Correlation values for Web-site activity parameters and course theory and case-study grades. 



Teams using the Web-site more frequently for reading the work of other groups did not score better in the 
grading of theory exam and case-study reports. Significant correlation is found between the quality of the 
contributions and the theory grade: -0.41 in 1998, versus + 0.55 in 1999. The negative correlation of 1998 
indicates that some groups good in theory did not perform well in the Web-site activity, which was also 
theory related. A student-panel interview confirmed that their motivation for the Web-site task in 1998 was 
low. With the adjusted set-up for 1999, the percentage of approved contributions is now a positive predictor 
for the theory grade. Further analysis showed that there is no significant correlation between the theory 
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grades and the case-study grades of the course. In the interview the instructors confirmed that transfer from 
reading theory to application of theory in problem-solving is not taking place as much as they would like. 



Discussion 

From our results it appears that both versions of the Web-site helped improve the discipline of reading 
theory articles. The revised 1999 set-up of the Web based "Theory Repository" was perceived as more 
helpful by students. This time substantial cross-group exchange of expertise took place. Although many 
factors have remained relatively stable over the years, a number of possible reasons for the differences in 
cross-group reading of contributions can be postulated: 

1. In 1998 the contributions consisted of sets of questions and answers. The 1999 summaries may be 
more helpful when students want to leam about an article without reading it. 

2. The pre-selection of the excellent articles in 1999 made this option more efficient for the students. 

3. The difference of 'control' by students: choosing themselves to read work of others or not (1999), 
compared to being forced to read others work and then rank it (1998). 

4. The improved user interface making the use of the Web-site more efficient. 

The combination of changes in instructional design with an improved user interface makes it difficult to 
identify the dominant cause of the improvement. However, based on discussions with students, we believe 
that the first explanation, a relevant and helpful task, dominates the students’ appreciation. Web-support 
can help organise these tasks in an efficient way. Although the Web-support thus activates the learning 
behavior of the students, students appreciate most highly those learning settings in which the students 
interact with their peer students and the instructors. However, we believe that these discussions are more 
fruitful when the students have been better introduced to the relevant theory. A correlation was found 
between the students’ Web-related activities and the theory examination results, but not between Web- 
related activities and the final case-study results. Transfer from the theory parts of the course to the case- 
study problem-solving tasks is thus not as straightforward as anticipated. 
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